Augmented, virtual and mixed-reality content selection and display

ABSTRACT

Systems, methods and techniques for automatically recognizing two or 3-dimensional real world objects with an augmented reality display device (smartphone or glasses etc.), and augmenting or enhancing the display of such real world objects by superimposing virtual images such as a still or video advertisement, an opportunity to buy, a story or other virtual image presentation. In non-limiting embodiments, the real world object includes visible features including visible security features and a recognition process takes the visible security features into account when recognizing the object and/or displaying superimposed virtual images.

CROSS-REFERENCES TO RELATED APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 16/893,236filed Jun. 4, 2020, now U.S. Pat. No. 10,997,418; which is acontinuation-in-part of U.S. patent application Ser. No. 16/565,234filed Sep. 9, 2019, now U.S. Pat. No. 10,699,124; and said U.S. patentapplication Ser. No. 16/893,236 filed Jun. 4, 2020 also claims benefitof U.S. Provisional Patent Application No. 62/966,503, filed Jan. 27,2020. Each of these prior applications is incorporated herein byreference for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

None.

BACKGROUND & SUMMARY

Augmented reality (“AR”) is the real-time superimposition of artificialimages onto real world images. This technology has been used for avariety of applications from handheld entertainment to heads-up displayson military jets. For example, a person using an augmentedreality-enabled display device (e.g., a smartphone with an AR app or ARenabled glasses) while shopping can be shown virtual advertisements,sale announcements, pricing information etc. superimposed onto images ofactual products they can purchase.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an example process flow diagram.

FIG. 1B shows an example branching process.

FIG. 1C shows an example area of recognition.

FIGS. 2A-2J show example items to be recognized.

FIG. 3 shows an example block diagram of a smart device and associatedsystem.

FIG. 4 shows an example non-limiting process flow diagram.

FIGS. 5A and 5B show an example augmented reality image superimposing avirtual image onto image captures.

FIGS. 6A and 6B show an example augmented reality image superimposing avirtual image (a video) onto an image capture of the front cover of a USpassport.

FIG. 7A shows an example capture of an image of a print advertisement ona smart device.

FIG. 7B shows an example area of recognition of the FIG. 7A imagecapture.

FIG. 7C shows an example augmented reality image including superimposedvirtual images (e.g., virtual shopping carts) onto the FIG. 7B imagecapture.

FIG. 8A shows an example image capture of a print advertisement.

FIG. 9A shows an example area of recognition of the FIG. 8A imagecapture.

FIG. 9B shows an example augmented reality image superimposing virtualimages (virtual shopping carts) onto the FIG. 8A image capture.

FIG. 10A shows an example image capture of a product trade dressincluding an area of recognition.

FIG. 10B shows an example augmented reality image superimposing avirtual image (e.g., a video of ingredients) onto the image capture ofthe FIG. 10A product trade dress.

FIG. 11A shows an example image capture of a real estate printadvertisement including an area of recognition.

FIG. 11B shows a closeup view of the FIG. 11A area of recognition.

FIG. 11C shows an example augmented reality image superimposing avirtual image (e.g., a listing of additional information concerning theproperty) onto the image capture of FIG. 11A.

FIG. 12 shows an example employment listing.

DETAILED DESCRIPTION OF EXAMPLE NON-LIMITING EMBODIMENTS

Because AR tends to drive higher levels of attention, one advantageoususe of augmented reality is superimposing advertisements and/orpurchasing information over product images. People who might ignoretraditional electronic advertisements may pay attention to an ARadvertisement. Same with traditional printed advertisements, which canbe used as a trigger to superimpose a digital overlay which then opensup new options like adding videos, live streams, buying options or otherinformation and content.

This technology provides systems, methods and techniques forautomatically recognizing two- or three dimensional real world objects(any kind of print or the package of a product) with an augmentedreality display device, and augmenting or enhancing the display of suchreal world objects by superimposing virtual images such as a still orvideo advertisement, a story, a tutorial, specific information or othervirtual image presentation.

In non-limiting embodiments, the real world object includes visiblefeatures including visible security features and a recognition processtakes the visible security features into account when recognizing theobject and/or displaying superimposed virtual images.

In the near future, when AR becomes really popular with any kind ofglasses or other viewing arrangement, we might also see at least apartial replacement of the smartphone. That means that we will see newuse cases that use AR or mixed reality to provide users or customerswith new information, any kind of content or buying options triggered bythe recognition of a print that could be a 2 or 3 dimensional object.For example, if you see a job advertisement, with your AR glasses or ahandheld device you can watch a related video or activate a call buttonto talk to somebody. When you see a travel advertisement, you point thecamera or other viewing device on this and you can watch a video of thedestination, a buy option or different dates etc. When you see anadvertisement of a product, you can decide to watch a related video, getother product information or buy it. This same scenario can exist with aproduct or service manual that could be the trigger of specificinformation that will allow you to understand how a product works bygiving you technical drawings etc. This same scenario could apply in thereal estate market where you can see a video of the object, specificinformation such as a neighborhood, floorplan, a call button to call theagent etc. This same scenario could apply with tickets where there is acity light as a trigger and when the AR kicks in, you can buy thisticket for a concert, a sport event etc. It could be that specificinformation about that concert or event can be provided to motivatecustomers to purchase. That could be videos or other specificinformation about players, a track record, etc. In one exampleembodiment, a device enables or allows superimposition of a digitaloverlay that leads to specific action such as providing specificinformation; providing videos, tutorials, or any kind of content;providing options to buy; or other options.

FIG. 1A shows an example non-limiting process. At 10, a printed,engraved, stamped or other object 102 such as a banknote, coin or ID(see FIGS. 2A-2J) is scanned to produce a digital copy. Such scanningcan be performed with a camera, a CMOS light sensor, a document scanner,or any other conventional device that creates a digital image of theprinted object.

The printed object may include visible features such as distinctiveimages of persons, buildings, etc. In the example shown, the printedobject further includes an example visible security feature Srepresented here as a vertical band. The scanned digital copy willinclude the distinctive image(s) as well as the visible security featureS.

At 20, the resulting digital copy of the printed object is uploaded intoa database, and enhanced to provide a selected associated overlaycomprising: (a) self-starting video(s) and/or video(s) that start whenyou physically press a button that has been programmed to show up on thesurface of a device like a mobile phone, Ipad etc., or activate thisactivity by voice, movements of your head and/or gesture; (b) button(s)that when selected have different interactive functions such as “tell astory”; and (c) interactive connection to an online shopping experiencee.g., to show a store finder (see 50). We can change the digital“overlay” quickly if needed (e.g., to change from one advertisingcampaign to another) or update content, price tags, videos or otherinformation.

To activate that overlay, we determine an “area of recognition” asindicated by a cross-hatched area (30). In one example embodiment, thearea of recognition is determined at least in part by the presence,position, dimensions and/or orientation of one or more security featuresS. For example, the area of recognition may be defined to exclude thesecurity feature S on the printed object. As an example, in the case ofa 20 Euro note (see FIG. 2E) including a vertically-oriented silverholographic band which appears differently depending on the viewingangle, the “area of recognition” may defined to exclude the silverholographic band so the system is able to recognize the object evenwithout a silver line. Similarly, the area of recognition in the case ofa $100 US banknote shown in FIG. 2C may be defined to exclude securityfeatures B, C and E since at least some of these features may appeardifferently depending on viewing angle (e.g., the ink color changes fromgold to green as viewing angle changes). Other security features such asserial numbers, bar codes etc. vary from one printed document toanother, so that using such indicia will result in no recognition of abanknote of the same denomination but having a different unique serialnumber and/or unique bar code (i.e., the purpose of the recognition maybe to recognize a particular kind of banknote, not that specificone-of-a-kind banknote). Excluding such security features from the areaof recognition makes recognition more robust because it is not subjectto such constraints.

The recognition of a specific area then triggers a digital overlay thatincludes defined features (video, information, content, shopping, etc.)that show up and can be used by a person to get information, see avideo, shop etc.

In another embodiment at 40, at least some security features S areincluded in the defined area of recognition. The object is recognizedonly if it includes the security feature(s). If there are visiblesecurity features, we can include them into our recognition program.When we upload the digital copy of a print, we can decide which area ofthe print is used for recognition. See FIG. 1C. When this area ofrecognition includes an example such as the silver line of a banknote,the note cannot be recognized when the silver line is missing. In thisembodiment, the security feature S is used as a feature of recognitionto increase the reliability of the recognition to recognize distinctivedocuments, while designing the recognition algorithm to take differentviewing angles into account. For example, it is possible to include theposition and orientation of a silver holographic band in an area ofrecognition without requiring the holographic contents of the band to berecognized. Similarly, it is possible to include the position of serialnumbers or bar codes in an area of recognition without requiring thespecific serial numbers or bar code contents to match.

Some embodiments provide plural overlapping areas of recognition for thesame object; one area of recognition may exclude certain securityfeatures and another area of recognition includes those securityfeatures. The plural different overlapping areas of recognition can beapplied sequentially or simultaneously to increase recognitionreliability. A voting algorithm can be used to select positive matches.

In example non-limiting embodiments, the database enables real timerecognition of an image captured by a user. For example, if the usercaptures an image of a 20 Euro note, a matching algorithm is used todetermine a positive match if the database contains a digital copy of a20 Euro note. In example non-limiting embodiments, the matchingalgorithm can include pattern recognition techniques such as describedin Conti et al, Multiresolution image registration, Proceedings,International Conference on Image Processing (IEEE 23-26 Oct. 1995);Hasanuzzaman et al, Robust and effective component-based banknoterecognition by SURF features, 20th Annual Wireless and OpticalCommunications Conference (IEEE 15-16 Apr. 2011); Doush et al, Currencyrecognition using a smartphone: Comparison between color SIFT and grayscale SIFT algorithms, Journal of King Saud University—Computer andInformation Sciences Volume 29, Issue 4, October 2017, Pages 484-492.

As will be understood by those skilled in the art, the database 20 couldbut need not contain the captured images themselves. For example, insome embodiments, the database might contain compressed or other featuresets used for comparing with captured photos, such feature sets forexample comprising a listing of coordinates and associated imagefeatures to thereby reduce storage requirements and increase recognitionspeed. Similarly, when a smart device captures an image, instead ofuploading the entire image it may analyze the image and upload acompressed format such as coordinates of pattern features. Since thepurpose of recognition in the example non-limiting embodiment is not todetermine whether the banknote or other printed item is authentic andgenuine, the matching/recognition standards can be significantly relaxedand thus quite different as compared to conventional banknotescanners/recognizers.

Some example embodiments use artificial intelligence and machinelearning to perform the matching. The training set consists of imagescaptured by various smartphones and other user devices. Otherembodiments use simpler pattern recognition algorithms based on colorand/or grey scale information in the area(s) of recognition. Any givenobject can have at least one and as many as N different discrete areasof recognition, where N is a positive integer.

FIG. 3 shows an example non-limiting system 200 including a user devicesuch as a smartphone 202, AR glasses, etc. which includes a camera orCMOS sensor 204, a processor 206, a memory 208, a display 210 and userinput/output devices 212 such as a touch screen, push buttons, etc. Theprocessor 206 communicates wirelessly or through a wire via a network214 with a server 216 which may store the database 20 and in someembodiments perform the image recognition. The app may compriseexecutable instructions that the device 202 downloads from the server216, stores in memory 206, and executes in response to user selection ofinput/output devices 212.

From a user perspective, as shown in FIG. 1A, after downloading an apponto their smart phone, smart glasses or other portable or non-portabledevice 202, the customer can point the device's camera onto a print,physical object or a digital copy on the screen of his tablet, computer,etc. (100; FIG. 4 blocks 302, 304, 306). This can be the camera of adevice like an Iphone or an external camera or a camera that is a partof AR glasses. This causes the device 202 to display a photographicallycaptured image on display 210 (see FIG. 5A, 6A). It is possible in someembodiments for device 202 to obtain this captured image some other waythan photographing an actual object 102, such as via network 214, e.g.,by downloading the image through Instagram or other photo sharingservice, by receiving the image via a text (SMS) message, by receivingthe image from another device in proximity (e.g., using Bluetoothpairing, etc.) or by other means.

Once the app has been activated to recognize the captured image, the appconnects with the database 20 on server 216 via network 214 and checksif this print is recognized (FIG. 4 blocks 308, 310). If recognition issuccessful (branch 110), an action is enabled (e.g., playing a video,telling a story and/or showing a picture, showing shoplinks or otherinformation) (FIG. 4 , blocks 312, 314). If recognition is unsuccessful(branch to 120), no action is enabled or a different action (“xyz”) maybe performed instead.

In one non-limiting example, the app causes the smartphone or otherdevice 202 to show a picture or video by anchoring it to the imagecurrently being captured by the device's camera 204 and displayed on thedevice's display 210. See FIG. 5B, 6B. If the user moves the deviceslightly to change the position or orientation of the printed item onthe device's's display 210, the picture or video will move with theimage of the printed item on the display as if it were glued orotherwise adhered to the printed item or were being displayed by theprinted item itself from a magic display window on the printed item. Forexample, the superimposed image can change position, orientation and/orperspective as the underlying real world object changes position,orientation and/or viewing angle in order to provide a photorealisticimage in which the superimposed image appear to be part of theunderlying real world image. See for example FIG. 6B. This augmentedreality feature provides a sense of wonder or surprise that is designedto capture the attention of the user. Instead of appearing to be a“flat” advertisement, the picture or window appears to be much moreinteractive and interesting and thus captures the user's attention. Theaugmented reality image can be animated and can be accompanied by audioand/or tactile sensations as indicated in FIGS. 5A, 5B, 6A, 6B.

In other embodiments, so-called “mixed reality” or “virtual reality” canbe used instead of or in addition to augmented reality. Generallyspeaking, “virtual reality” presents the user with an image of anentirely virtual/non-real world. The posture (orientation and/orposition) of the device may be sensed (e.g., using inertial sensors suchas accelerometers and/or gyrosensors; magnetic position and/ororientation sensors; RF beacon sensors; optical sensors providing anoptical “flow” image analysis; and other known conventionaltechnologies. Different portions of the virtual world may be displayedbased on such sensed posture.

Other embodiments can use so-called “mixed reality” (MR) that mixesvirtual and real scenes. Such an MR experience is one where the user isplaced in an interactive setting that is either real with virtual assetaugmentation (augmented reality), or virtual with real-worldaugmentation. See Ohta et al, Mixed Reality: Merging Real and VirtualWorlds (Springer Publishing 2014); Milgram et al, A Taxonomy of MixedReality Visual Displays, IEICE TRANSACTIONS on Information and SystemsVol. E77-D No. 12 pp. 1321-1329 (Dec. 25, 1994); Hughes et al, Mixedreality in education, entertainment, and training, IEEE ComputerGraphics and Applications (Volume: 25, Issue: 6 November-December 2005).

In more detail, a user can aim the camera of a smartphone, smart glassesor other electronic device at any two-dimensional or three-dimensionalreal world object or a likeness of such an object. One example type of atwo-dimensional object could be a portable, wallet-sized planar objectsuch as a banknote, driver's license, passport, printed advertisement,real estate advertisement or other official or unofficial identificationdocument examples of which are shown in FIG. 2A-2I. Other examples ofsuch objects might include for example a real estate advertisement, ahuman resources want ad, and objects related to a shopping experience.Using the technology with AR Glasses instead of a mobile phone means,that you do not touch a button of the superimposed digital image on yourscreen but press either a virtual button to shop or get information orjust say that you want to press a specific button or say that you wantto see a video or say that you want to buy a merchandise that is offeredwithin the digital overlay,

For example, the object as shown in FIG. 7 may comprise a real estateadvertisement which can show one or several houses, lots, commercialbuildings, private properties etc. An example area of recognition mayconstitute a single advertisement or a portion of an advertisement (seeFIG. 8 ). Image recognition may be used to recognize the particular realestate property, and such recognition may be used to select and initiatedisplay of a media item that provides more images and informationconcerning that property.

These information can be floor plans, videos of the property, contactdata of the agent, the lot size, drone videos, the actual price tag orany other information that ate relevant for the seller, the buyer or theagent.

The same with any shopping or travel advertisements where you define anarea of recognition which could be a part or parts of that advertisementor even the full page to trigger a superimposed image that containsvideos, shoplinks or other buttons that provide other content orinformation and allow to buy merchandise, travel related services(flight, hotel, tour, cruise, travel package, insurance and/or watch,read related information.

Another example for a use case is related to packaging, where you definean area of recognition on a 3 dimensional object. After a match in thedatabase you can add product information, videos, or any other contentor buy the specific merchandise.

As another example, the object shown in FIG. 9 is an employmentadvertisement. Image recognition processes may be used to recognize theadvertisement. Successful recognition can be used to select and causedisplay of a media item that provides more information and imagesconcerning the employment opportunity. For example, there is a stronguse case where you point the camera on a standard HR advertisement andthen you see, for example, the CEO or the HR Manager of the companydescribing that specific job.

As yet another example, the object may comprise the face, head or bodyof a person or animal. Image recognition processes may be used torecognize the person or animal. Successful recognition can be used toselect and cause display of a media item that provides more informationand images concerning the recognized person or animal.

Yet another example uses an object comprising a shopping item such as aparticular product or service offering. Image recognition processes maybe used to recognize the product or service offering. Successfulrecognition can be used to select and cause display of a media item thatprovides more information and images concerning the recognized productor service offering, or buying options

In one example, such objects may be made of paper, plastic and/or othermaterials, and may be rigid, flexible and/or foldable. They may be ofstandard sizes such as 2.61 inches wide and 6.14 inches long; 120 by 62millimetres (4.7 in×2.4 in) to 160 by 82 millimetres (6.3 in×3.2 in);the ID-1 format size of 85.60×53.98 mm (3⅜ in×2⅛ in approx.) and roundedcorners with a radius of 2.88-3.48 mm (about ⅛ in); ID-3 standard sizeof 125×88 mm (4.921×3.465 in); or other size. See for example ISO/IEC7810:2003. The object may be two-dimensional or three-dimensional.

One example such object comprises a banknote such as a US dollar bill,US five dollar bill, US ten dollar bill, US twenty dollar bill (see FIG.2A), US fifty dollar bill (see FIG. 2B), a US one-hundred dollar bill(see FIG. 2C), a 20 euro note (see FIG. 2D), a 50 euro note (see FIG.2E), or any other currency banknote. Other example objects compriseofficial government-issued identification such as a government issuedpassport (see FIG. 2F), passport card (see FIG. 2G), driver's license(see FIG. 2H, 2I), coin (FIG. 2J) or other official or unofficialobject.

Such two-dimensional objects as described above often are protected byany or all of the following visible security features:

holograms such as portrait window(s)

woven three-dimensional security ribbon(s)

distinctive image pattern(s) such as the EURion constellation

glossy stripe(s) that reveal additional information when viewed on anangle

watermark(s)

raised printing

ultraviolet ink (which reveals additional image information when viewedunder ultraviolet light)

thermochromatic ink (the color of which is dependent on temperature)

color changing or color shifting ink (the color changes depending onviewing angle)

see through indicia with complementary front and obverse portions

microprinting (high resolution printing)

unique printing fonts

embedded security threads

perforations

plastic, cotton fiber or other material matted or other texturedsurface(s)

barcodes

serial numbers

foil stamps

photographs including clear window photographs

unique metal compositions (for coins) such as bi-coloration

other.

Example non-limiting recognition processes as described above canexclude such security features, or may take them into account or usethem as part of the recognition process. However, since the purpose ofthe recognition is not to authenticate the photographed item as beinggenuine, the recognition/matching algorithm is quite different from onesthat are used for banknote or ID authentication. In example non-limitingembodiments for example, it is desirable that matching occurs based onphotographing a copy (e.g., displayed on a tablet screen or the like)and not just an original of a banknote, ID or the like. Thus, thematching will achieve positive results based on counterfeit(inauthentic) banknotes or IDs. However, the matching/recognition isrobust in being able to detect different banknote denominations (e.g.,10 euro note versus 20 euro note versus $100 US bill etc.) and thusprovide different overlays depending on which banknote is recognized.

In still other embodiments, the two-dimensional or three-dimensionalobject could comprise a political campaign poster, billboard, flyerpackage, floor stand, table stand, pen with advertisement or otherprinted material or 3 dimensional object with print and the overlaycould provide a campaign song, slogan, speech, story, video or othercontent. Other types of campaigns could include advertising campaigns,company logos and printed advertising or any other printed campaign fromcompanies or other entities.

As discussed above, other embodiments can employ a two-dimensionalobject comprising a real estate advertisement, a job advertisement, aproduct advertisement, a service advertisement, or any otherpresentation of an item, entity or organization that a user may wish toobtain more information concerning.

In example embodiments herein, the two-dimensional object is not (anddoes not need to contain) a so-called “AR marker” or 2D bar code and isotherwise not specially designed or intended to be recognized by anaugmented reality display device.

Example 1

FIG. 5A shows a smart device 202 the camera of which has captured animage of a $20 bill. Once the app recognizes the captured image 402 asdescribed above, it displays a corresponding advertisement 404 as shownin FIG. 5B. The advertisement may be superimposed over the capturedimage 402.

Example 2

FIG. 6A shows a smart device 202 the camera of which has captured animage of a US passport. Once the app recognizes the captured image 406as described above, it displays a corresponding video 408 (as shown inFIG. 6B) that is superimposed onto the captured image 406. If the userrotates the smart device 202 to change the position and/or orientationof the captured image 406, the corresponding video 408 will also changeits position and/or orientation by the same amount. In some embodiments,the video 408 may be displayed with controls such as play/stop,fast-forward, rewind and mute that the user can press on the touchscreen display 210 to allow the user to control playback of the videoand/or accompanying audio.

Example 3

Instead of a video, FIG. 6B could display a superimposed control barthat allows the user to play an audio story associated with the capturedimage 406.

Example 4

FIG. 7A shows a captured image of a print advertisement. FIG. 7B showsan example area of recognition. FIG. 7C shows a superimposed virtualimage (shopping carts) on the FIG. 7A captured image to provide avirtual or mixed reality composite image.

Example 5

FIG. 8A shows an example of a captured image of a print advertisement.FIG. 9A shows an example area of recognition. FIG. 9B shows asuperimposed virtual image (a shopping cart) on the FIG. 8A capturedimage to provide a virtual or mixed reality composite image.

Example 6

FIG. 10A shows a captured image of a product package with a defined areaof recognition. FIG. 10B shows a superimposed virtual or mixed realitycomposite image providing a video of ingredients.

Example 7

FIG. 11A shows a captured image of a print advertisement for real estatelistings. FIG. 11B shows a closeup of an area of recognition. FIG. 11Cshows a superimposed virtual image on the FIG. 11B captured image.

FIG. 12 shows an example captured image of a human resourcesadvertisement. This captured image can have an area of recognition andcan have a digital image superimposed thereon to provide an augmented ormixed reality image on a smart device such as a smartphone, smartglasses, etc. In some embodiments, the display process may compriseprojecting the digital overlay directly or indirectly onto a user'sretina. Such techniques would allow the user to view the digital overlayon a screen (e.g., smartphone) and/or on glasses that are used as ascreen and/or independently of any screen viewing (i.e., the user couldsimply view the real world and the digital overlay would become part oftheir field of view based on such projection directly onto theirretina).

All printed publications cited above (including the ones shown in FIGS.2A-2J) are hereby incorporated by reference as if expressly set forth.

While the invention has been described in connection with what ispresently considered to be the most practical and preferred embodiments,it is to be understood that the invention is not to be limited to thedisclosed embodiments, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

The invention claimed is:
 1. A method of presenting image information toa user of a portable device having a portable camera and a portabledisplay, the portable device being of the type that is freely movablerelative to a printed object having indicia printed thereon so capturedimages of the printed object change in position and/or orientation, themethod comprising; specifying an area of recognition that is defined toexclude at least one indicium on the captured images of the printedobject, for use in recognizing a pattern associated with the printedobject from the captured images based on color and/or grey scaleinformation in the area of recognition; and specifying a media item forpresentation to the user in response to recognition of the pattern, thespecified media item being configured to be superimposed onto thecaptured image or an image derived therefrom displayed on the portabledisplay.
 2. The method of claim 1 wherein the specified media item isconfigured to be superimposed for augmented reality, mixed realityand/or virtual reality.
 3. The method of claim 1 wherein the specifiedrecognition area is configured to be used to recognize a two-dimensionalor three-dimensional printed object with print on it.
 4. The method ofclaim 1 wherein the specified recognition area is configured to be usedto recognize characters printed on the object.
 5. The method of claim 4wherein the object comprises a printed advertisement.
 6. The method ofclaim 5 wherein the media item comprises a digital overlay that leads tospecific action selected from the group consisting of providing specificinformation; a video, tutorial, or any kind of displayable content; oran option to buy.
 7. The method of claim 1 wherein the portable displaydevice is handheld or worn.
 8. The method of claim 1 wherein the printedobject comprises a product or service manual and the method furtherincludes triggering of specific information that will allow a user tounderstand how a product works by providing technical drawings.
 9. Themethod of claim 1 wherein the selected media item comprises a callbutton.
 10. A method of providing a message to a user of a portabledevice comprising portable camera and a portable display, the portabledevice of the type that is freely movable relative to a billboard,package, object with print on its surface, flyer or other printedrepresentation so images of the billboard, package, object with print onits surface, flyer or other printed representation captured by theportable camera change in position and/or orientation, and displayscaptured images of the billboard, package, object with print on itssurface, flyer or other printed representation on the portable display,the method comprising: defining an area of recognition to exclude atleast one indicium within the captured images, and area of recognitionbeing defined to enable the portable device or a server operativelyconnected to the portable device to recognize the billboard, package,object with print on its surface, flyer, or other printed representationbased on grey scale and/or color values within the area of recognition;and providing an augmented reality, virtual reality or mixed realityoverlay for being overlaid by the portable device onto the displayedimage, the provided overlay providing an adjunct message associated withcontent of the billboard, package, object with print on its surface,flyer or other printed representation, the overlay being configured toenable the portable device to move the overlay on the display inresponse to changing position and/or orientation of the captured imagesof the billboard, package, object with print on its surface, flyer, orother printed representation to simulate the overlay adhering to thebillboard, package, object with print on its surface, flyer, or otherprinted representation or being displayed from a display window on thebillboard, package, object with print on its surface, flyer, or otherprinted representation.
 11. A server for presenting image information toa user of a portable device having a portable display and a portablecamera that is freely movable relative to a printed object havingindicia printed thereon so images of the printed object captured by theportable camera change in position and/or orientation, comprising: aprocessor in communication with the portable camera and the portabledisplay, the processor being configured to perform operations including:using an area of recognition that is defined to exclude at least oneindicium on images of the printed object captured by the portable camerato recognize a pattern associated with the printed object based on colorand/or grey scale information in the area of recognition; selecting amedia item in response to the recognized pattern; and controlling theportable device to superimpose the selected media item onto the capturedimage or an image derived therefrom displayed on the portable display,including moving the superimposed selected media item in response tochanging position and/or orientation of the captured images of theprinted object.
 12. The server of claim 11 wherein the server enables atleast one of augmented reality, mixed reality and virtual reality. 13.The server of claim 11 wherein recognizing comprises recognizing atwo-dimensional or three-dimensional printed object with print on it.14. The server of claim 11 wherein the processor is configured torecognize characters printed on the object.
 15. The server of claim 14wherein the object comprises a printed advertisement.
 16. The server ofclaim 11 wherein the selected media item comprises a digital overlay,and the processor is further configured to receive a user selection ofthe overlay and to use the user selection to trigger action selectedfrom the group consisting of providing specific information; a video,tutorial, or any kind of displayable content; or an option to buy. 17.The server of claim 11 wherein the portable device comprises a handhelddisplay device or smart glasses.
 18. The server of claim 11 wherein theprinted object comprises a product or service manual and the processoris further configured to trigger presentation of information that willallow a user to understand how a product works by providing technicaldrawings.
 19. The server of claim 11 wherein the selected media itemcomprises a call button.